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ABSTRACT 

Protocol analysis, a technique that uses people's 
verbal reports about their cognitions as they engage in an assigned 
task, has been used in a number of applications to provide insight 
into how people mentally plan, assess, and carry o^it those 
assignments. Using a system of networked computers where actors 
communicate with each other over terminals and simultaneously talk 
aloud their thoughts, researchers can gain insight into people's 
cognitions during conversations. Participants converse over a 
computer-mediated channel, while simultaneously providing verbal 
protocols about their thoughts. It takes participants at least two 
conversational sessions before they begin to feel comfortable using 
and interacting on a mediated system. Examples from protocol data and 
past research illustrate how plans, evaluation, attributions, and 
shared experiences are activated during conversations. Advantages of 
the design are: it offers a new sort of data; data collection is easy 
once conversationalists are acquainted with the system; and there is 
much potential for investigating questions about cognition and/or 
interaction. Limitations include: the extent to which verbal 
protocols reflect cognition specific to the conversation; the sheer 
amount of time and effort to transcribe the videotapes; and questions 
regarding participants' varying ability to provide oral protocols. 
Analysis of protocol concurrent to conversation through a 
computer-mediated channel offers a unique opportunity to map 
cognition during interaction. (Contains 11 references.) (RS) 
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Abstract 

Protocol analysis (Ericsson & Simon, 1984), a technique that uses people's verbal 
reports about their cognitions as they engage in an assigned task, has been used in a 
number of applications to provide insight into how people mentally plan, assess, and 
carry out those assignments. Likewise, protocol procedures can be used to examine 
conversation. Using a system of networked computers where actors communicate 
with each other over terminals and simultaneously talk aloud their thoughts, 
researchers can gain insight into people's cognitions during conversations. This 
paper describes the method of observation, illustrates some findings, and identifies 
the investigative potential and limits of the method. 
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Protocol Analysis as a Method for Analyzing Conversational Data 

One of the most enduring beliefs in the field of communication is that the 
processes of forming and expressing messages begins with cognition. While scholars 
may argue over the level of control and consciousness actors have over these 
processes (Kellerman, 1992), or the extent to which conversation can be considered a 
"mindful" activity (Bavelas & Coates, 1992), most agree that the link between what 
goes on in people's heads, what comes out of their mouths, and what they perceive to 
be happening in their environment is more than randomly fired neurons. To be 
sure, in viewing the mind as an integral part of individual and social conduct (Mead, 
1934), as opposed to the brain , one is allowed to make inferences about the so-called 
human condition, and how this condition differs from that of non-humans. 1 

Scholars studying associations between the mind and communication generally 
take one of three approaches. Some experimentally manipulate a variable related to 
cognition and make inferences about cognition based on the influence of that 
variable on participants 1 behaviors. Others use more descriptive approaches to draw 
conclusions about speakers' cognitions from observations of natural or reconstructed 
texts. Still others take a retrospective approach and ask people to recall and describe 
their thoughts following interaction. Each of these yield important data concerning 
cognition and communication, yet they share a common limitation. Each is unable to 
reveal participants' concurrent cognitions— the thoughts people have simultaneous 
with the communication event. 

Our interests are to help fill this topographical void. Since there is evidence 
associating cognitive processes to interaction before and after a communication 
event, we can expect there are associations during the event as well. The obvious 

1 The mind is a complex, multi-levelled, abstract construction. While the mind includes levels 
of unconsciousness that also explain the uniquely human outlook on life, self, and other, we speak 
more to those properties that can be used to account for how humans socially interact. 
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problem is an inability to actually view the abstract entity of the mind during 
conversations Pencil-paper measures that ask people to select or write messages 
they would employ in hypothetical situations may tell us how thoughts arrange talk 
prior to interaction. On the other hand, having people orally describe what is going 
on in their minds while they complete a pencil-paper measure may tells us more 
about the "on-line" cognitive activity during interaction than the measure itself. 

Protocol analysis, (Ericsson & Simon, 1984), a technique that uses people's verbal 
reports about their cognitions as they engage in an assigned task, has been used in a 
number of applications to provide insight into how people plan, assess, and carry out 
those assignments. Since it is impossible for people to provide verbal reports during 
their conversations with another, we've devised a means whereby actors converse 
with each other through a computer-mediated channel, while simultaneously 
providing the researcher with verbal descriptions of their cognitive activity. The 
remainder of this paper will describe our method of gathering protocol concurrent 
to conversation, illustrate some collected data, and discuss the potential and limits of 
this method. Throughout this paper, we refer to mediated conversations as simply 
conversations, even though there are physical differences between the two. 
Setting Up A System and Gathering Concurrent Protocol Data 

Ericsson and Simon (1984) summarize that in fields such as cognitive 
psychology and educational research, participants are often asked to think aloud as 
they compose a brief paragraph, solve a math problem, or complete some other task. 
What they say as they work through a task helps scholars understand the cognitive 
mechanisms involved in that task. While the verbal descriptions themselves may not 
be the actual processes that occur, those self-descriptions may be used to tesr and 
develop theory on cognition and task fulfillment. 

Using such a method for conversations, however, is problematic given that the 
channel required to provide verbal descriptions of thought is occupied with 
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providing speech to a conversationalist. Unlike the mind, which because of its 
multi-levels of complexity can perform many simultaneous functions, the mouth can 
only perform one function in a given time space. At issue then is how to leave the 
vocal channel available for providing descriptions of cognition but attain some sense 
of conversation structure not available from alternative mediums like letter writing. 

This issue was addressed by Daly et al.( i989) who devised a method that uses two 
separate channels for communication: one for the oral protocols, and one for the 
conversations. Participants converse over a computer mediated channel, while 
simultaneously providing verbal protocol about their thoughts. Both protocol and 
conversations are recorded for analysis. Since the conversationalists' computer 
terminals are in separate rooms, neither participant has access to the others 1 
verbalized thoughts during the interaction. 

Minimally, the design requires: two computer terminals placed in separate 
locations, but linked to a common networking system; two video cameras; two video 
recording units; two audio mixers; and two microphones. Networking systems and 
software programs vary, but electronic interact software that process and send 
information on "real time" are the most desirable. The research facilities for the 
Communication Studies Department at the University of Iowa, for instance, use 
NetWare by Novell Inc. on an IBM compatible system. The system runs a program 
specially written for collecting concurrent protocol data (see Daly, Vangelisti, Weber, 
Maxwell, & Neel, 1989). That program allows all messages to be sent as they are being 
entered by a user. Currently, there are four terminals on line, although the program 
is capable of serving five users at the same time. 

Conversationalists are led into separate rooms, each equipped with a table, chair, 
keyboard, monitor, camera, recorder, sound mixer, and microphone. Separating 
participants allows each to provide oral protocols that reflect their thoughts without 
worrying how those thoughts will affect their partner. The keyboard is placed 
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directly in front of the participant, while the camera is placed just behind, focussed 
on the terminal screen. Camera obstructions are avoided by placing the terminal 
screen a little to the left or right of the conversationalist, rather than directly in 
front of him/her. Although most cameras are equipped with a microphone, we've 
found them to be of little use, especially since many utterances of protocol are 
mumbled under breath. External microphones then are hooked up to a sound mixer 
and fed into the recording unit. Clip-on microphones have been most effective, but 
directional table microphones suffice. 

Depending on the research question, participants are instructed to describe 
their thoughts as they converse with another over the computer terminals. As actors 
converse, each camera unit records the text as it appears on the respective screens, 
and verbalized thoughts as they are uttered throughout the interaction. Recording 
the text rather than the participant allows the researcher to identify where oral 
protocol and mediated communication meet as well as diverge. Videotapes are then 
transcribed into conversation (text as it appears on screen) and protocol (recorded 
utterances). 

While the actual means of gathering protocol data is much the same, variance in 
interaction programs give the researcher more or less flexibility. For instance, the 
software used at the University of Iowa allows users to access another by "calling" 
their ID number, and having that other "pick up". Each users 1 screen is split into 
five areas, one area per user. As many as five users can participate in a conversation 
at the same time. Each area contains an ID for each user so that all on-line know who 
is "speaking." Entered text appears within the area of that user. Users must specify, 
then, who they are talking to when conversations are open to more than two 
members. The program is also able to produce a printed copy of the com ?rsation as 
it entered and was processed by the network server, providing accurate points of 
turns and "speaking" time. 
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The software program processes and sends information to all on-line screens as 
it is entered by the user. Unlike most E-mail programs, then, it is not necessary to 
produce an entire text before sending it. Likewise, a respondent, upon viewing the 
text as it is entered by another, can "interrupt" as well as "talk-over" while the other 
is still entering text. Hence, interaction on the system occurs in "real time" since, 
like talk, messages are immediate and non-retractable. 

Research questions underlying investigation greatly influence how verbal 
protocols are gathered. Participants may be asked to provide specific information or 
global descriptions about their thoughts. We've found that by stating our purpose as 
a global one we receive the greatest amount and range in oral protocol (e.g., "Simply 
to record whatever is on your mind as you talk to another over the computer"). More 
specific protocol requests tend to produce less protocol overall (e.g., "To record the 
attributions you make about another's statements. An attribution is..."). 

To train participants, we have the pair watch a 20 minute video that illustrates 
cognition during conversation (a scene from "Annie Hall"), explains the method as a 
means to meet our research needs, and demonstrates the actual method by showing 
one person's side of the mediated conversation. After viewing the tape, participants 
are shown the entire system to clarify any questions about the method, and to 
alleviate any fears regarding privacy. Unlike other studies of conversations, many 
participants want to make sure that their partner or others will not have access to 
their thoughts. After going through step-by-step instructions with each person on 
how to use the system, we run participants through a 20 minute practice session. 
The total amount of time required for training, completing consent forms and 
answering questions, is about 90 minutes. 

Although some of our data involve a confederate conversationalist, many of our 
studies are of interaction between pairs who have been in some type of relationship 
for over two years. Because concurrent protocol is generally information 
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unavailable to others, extra steps to maintain confidentiality are taken. First, 
participants are given a computer terminal ID to use and store information. That ID 
is used on accompanying diaries, logs, and questionnaires. Second, while it is 
sometimes necessary to monitor the conversations to ensure that the system is 
working appropriately, researchers and assistants do not monitor the protocol of the 
conversationalists. Third, conversationalists have the opportunity to stop or erase 
recorded data at any time. Fourth, assistants involved with collecting data are not 
allowed to view or transcribe those video tapes. 

It takes participants at least two conversational sessions before they begin to 
feel comfortable using and interacting on a mediated system. In fact, most pairs find 
the medium to be a novel experience. The extent to which participants feel 
comfortable or are able to verbalize their thoughts, however, seems to be a more 
personal matter. Initially, we had a research assistant in the room with each 
conversationalist, prompting them to verbalize what was on their mind. Obviously, 
this produced anxiety for some participants. We now place a small tape player near 
the computer terminal that randomly plays messages such as, "What are you 
thinking about?" or "Keep talking aloud" every 30-40 seconds. After the two 
sessions, participants begin to tune out the taped message and produce protocol at 
their own ease. 

While we initially attributed the matter of minimally produced protocol to 
cognitive overload, there does not seem to be a clear relationship between how easy 
actors find the computer system to use and how easy they find it to verbalize their 
thoughts. Nor does there seem to be a relationship between how at ease actors are in 
conversing and how easy they find verbalize thoughts. This matter in and of itself 
poses interesting questions regarding cognition, complexity, and verbal ability. 
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Samples of Collected Data 

Analysis of protocol concurrent to interaction is applicable to any number of 
communication events. However, our focus here is to illustrate the type of data this 
method may provide to help inform us of the cognitive processes related to everyday 
conversation. These examples were drawn from our protocol data and past research 
to illustrate how plans, evaluations, attributions, and shared experiences are 
activated during conversation. In every case, the computer mediated conversation is 
displayed in the left column; the protocol is in the right. Text that is contained 
within parentheses represents statements that the participant is reading from 
his/her computer screen. Participants 1 reports of their thoughts are underlined. 
Plans 

This excerpt illustrates conversational planning. Plans are projected actions based 
on generalizations from repeated similar experiences. In this case, the generalized 
scene is initiating a relationship with a stranger. After S notes her name and asks 
about her partner's class schedule, she begins to plan what she will say after her 
partner responds. This plan is not articulated to S f s conversational partner, but is 
verbalized in her protocol. 



CONVERSATION 

S: My name is Leslie. I know 
someone named Holly in my Spanish 
class, do you take Spanish? 



PROTOCOL 

S: Okay, (my name is Leslie, I know 
someone named Holly in my Spanish 
class. Do you take Spanish?) So now, 
well, I'm just waiting to hear what she's 
going to say. If she savs ves then I guess 
Til probably ask her probably her major 
or try and find something else we have 
in common. 



Evaluations 

During conversation, listeners sometimes evaluate what their partner says. These 
evaluations prompt a speaker to question whether or not to continue the direction of 
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conversation. That decision is often dependent on the attributions made about the 
speaker or utterances. In the example that follows, a confederate (L) has just 
disclosed some relatively intimate information (concerning her family's financial 
status) to M. M responds in his protocol (but not in conversation) with an evaluation 
of L's statement, noting that the utterance was not appropriate ("she's not supposed to 
say this"). 



Attributions 

Attributions are causal statements that assist in making sense of a situation, event, or 
behavior. Generally, attributions are made of conditions that a person finds positive, 
negative, or confusing. In the following excerpt, the participants (A & B) have been 
discussing a third party (Sue). Just prior to the excerpt, A appears to become agitated 
that this third party "went out 1 ' the previous evening. In the conversation, B 
questions A's agitation by asking, "Wasn't Sue supposed to be going out or 
something?" In her oral protocol, however, B concurrently makes a number of 
attributions about A's behavior. These are attributions that B chooses not to express 
in the conversation. 



CONVERSATION 



PROTOCOL 



L:...and my parents are going to 
need some financial help. 



M: I'm sure. Oh wow. Oh. I'm sure 
she's telling me this and I haven't 
even met her yet. What am I supposed- 
I feel bad but God, she's not supposed 
to say this the first time. I mean, I can't 
even see her face . 



CONVERSATION 



PROTOCOL 



B: Wasn't Sue supposed to be going 
out or something? 



B: (Wasn't Sue supposed to be going 
out or something). 



mavbe she TSuel was iust joking. 



You're taking things so literally that 
you do this computer thing because 
vou don't even see it as emotion.. .you 
iust take what they say as what . 
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After contrasting her behavior with behaviors previously described by her 
conversational partner, G. makes an attribution about her own behavior. Again, this 
is an attribution she chooses to withhold from her partner. 



Shared Experiences 

One factor that is often used to differentiate conversation in interpersonal from non- 
interpersonal relationships is the use of mutual knowledge (Planalp & Benson,^1992) 
and shared experiences to process and anticipate interaction. Unlike generalized 
scenes which can be used to explain how conversations are organized and produced 
(Kellerman, Broetzmann, Lim, & Kitao, 1989), the shared experience is a unique and 
specific event that forms some locus of meaning for interpp^sonal pairs. In this 
example, M & B have been in a romantic relationship for over two years. M is 
reminding B of their plans to go pick a pumpkin out for Halloween. M begins her 
sentence which prompts B to recall the shared experience. B anticipates what M is 
going to say, and accurately finishes her sentence. In the following two examples, 
conversation utterances are spaced to provide you the reader with a sense of time. 
Note how B's protocol is fully verbalized before M can get five words out. 



CONVERSATION 



PROTOCOL 



G:... Chris and I never do things 
like that together, but I'm not 
very romantic. 



G: (never do things like 
that together) but I ' m not that 
type of person . (I'm not very 
romantic) Yuk. I'm not thai 
type of person... 



CONVERSATION 



PROTOCAL 



M: Don't f orget 

to bring the 



B: (Don't f) orget to bring the 
pumpkin knife like you did last year! 
{laughter J I knew she'd bring that up ! 
{laughter} (Ha ha Did you 
think I forgot?) No. 



pumpkin knife like you did 
last year... Haha...Did you 
think I forgot? 
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As revealed in her own protocol, M's utterance is more than just a randomly produced 
statement. Her own thoughts reflect the same experience. What is very interesting 
in this case is that both participants are thinking of the same incident at the same 
time, rather than B's thoughts being activated after M's utterances. 



Potential and Limits of the Method 

Before discussing the potential and limits of this method, our use of it must be 
put into proper focus and context. The focus is cognition, the context is conversation. 

Although the method may not produce interaction which precisely mirrors the 
structure and form of conversation, there are advantages of the design. First, it 
offers a new sort of data. Most generally, it provides researchers with a means of 
observing and analyzing the thoughts that people have as they engage in a 
communication event. While these verbal reports may not assess all, or even most of 
the cognitions that occur, they very likely tap those thoughts that are most salient or 
available. Further, because participants report on their cognitions as they occur, 
researchers do not need to make judgements or inferences about the nature of those 
cognitions based on respondent behavior. Instead, the protocols provide a ready 
interpretation of cognitive processes associated with particular behaviors. 

Second, while getting started is a time consuming task, collecting data is rather 
effortless once conversationalists are acquainted with the system. Most of our 
conversational pairs looked forward to their weekly interaction sessions and 
reported being satisfied with their conversations even after six meetings. To be sure, 



CONVERSATION 



PROTOCOT 



M: Don't forget to bring 
the pumpkin knife like 
did last year. ..Ha ha.. .Did 
you think I forgot? 



M: I wonder if he thinks I forgot about 
him not bringing the knife last year ? 
{laughter} (Did 
you think I forgot?) 



13 



Protocol Analysis 13 



participants generally found the idea of collecting thoughts during conversation to 
be interesting and offered additional information regarding their thoughts and 
conversations once the studies were over. 

Third, there is much potential for investigating questions about cognition 
and/or interaction. For example, how do conversationalists manipulate codes to 
overcome interaction constraints inherent in the system? In one case, a "speaker" 
deleted his entire entry at the completion of his turn so that his partner would not 
"read too much" in to what he was saying. In another case, a "speaker" entered 
jibberish to let his partner know that he was frustrated and was no longer 
"listening" to her statements. In still another case, a speaker typed a single key 
consecutively to symbolize that she was pausing, and had not yet completed her turn. 
In many cases, conversationalists capitalized all words to express anger. 

Concerning cognition, more work can be done on the attributions actors provide 
for their own or others 1 behavior. For example, what reasons can "speakers" provide 
for not choosing to disclose or express information to another at the time of 
conversation? As illustrated by Vangelisti, Miller, and Aleman (1992), actors often 
choose to express some attributions, and withhold others during conversations. Past 
work has provided taxonomies of reasons people report for not expressing their 
thoughts to another, yet how these correlate with reasons made at the time of the 
conversation remain unknown. 

Relatedly, little is known of how people process emotion during conversation. 
People's emotive responses vary with a number of interactional and interpersonal 
factors (see Bowers, Metts, & Duncanson, 1985). Yet, aside from the study of neural 
firing, physiological arousal, and vocalic-noverbal cue during deception, little has 
been done to investigate people's emotive responses during conversation. There 
seems to be much to gain then by mapping how people describe their emotions as 
related to messages received, sent, anticipated, and recalled. Such study should keep 
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in mind, however, the differences in sensory, intellectual, and emotional biases of 
media, and how those biases may impact user response. 

As with any method, there are several limitations in collecting and analyzing 
protocol data. The first is the extent to which verbal protocol reflects cognition 
specific to the conversation. That is, while the mind is capable of performing many 
simultaneous functions, the production of verbal protocol itself is a behavioral 
activity influenced by cognitive processes. Participants may then attempting be 
addressing two distinct but simultaneous goals: communication with a partner, and 
communication with the researcher. The extent to which participants tailor their 
oral protocol for this or other purposes at this point is unknown. Researchers must 
further investigate, then, the validity and generalizability of verbalized thoughts 
made during conversation, mediated or otherwise. 

The second is the sheer amount of time and effort required to transcribe 
videotapes for analysis. Each tape contains two sets of data: the conversation in 
textual form; and the protocol in verbal form. Transcribing the conversation is a 
tedious task of reading text as it was recorded off the terminal screen. Transcribing 
the protocol is a tedious task of wading through mumbled utterances. Additionally, 
aligning protocol with the conversational text can be a tiresome effort, especially if 
the research question requires a careful and accurate matching of text as it appears 
on the terminal screen with the uttered thoughts. This point is even more dramatic 
if both conversationalists 1 thoughts are to be compared to the appearing text. 

Finally, there are a number of questions regarding participants 1 varying ability 
to provide oral protocol. As stated, whether or not a person feels comfortable or is 
able to verbalize their thoughts seems to be an individual rather than situational 
factor. Cognitive overload and communicative demand may be one reason for these 
differences. Excessive involvement in the conversation may be another. Concern 
for privacy may be still a third. The point is, like the variables that influence 
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participation in naturally occurring, face-to-face conversation, it is difficult to 
identify variables that affect the production of protocol, or that interact with related 
conversation variables. 

Clearly, computer-mediated conversations are not identical to face-to-face or 
even telephone conversations. Aside from the absence of nonverbal and vocalic 
information, computer-mediated codes differ from verbal messages in physical and 
symbolic form, both of which create biases of that medium. The physical form of 
computer mediated interaction is spatially biased (Innis, 1951). Despite that the 
software program processes information in "real time" and thereby makes messages 
"non-retractable" the second they are entered, those messages are conveyed in a 
literate form. Unlike oral messages, written code allows a receiver to "go back" and 
re-read what has already been "stated." Hence, there is less reliance on memory. 
Further, because literate messages remain visibly "frozen" in the context of the 
preceding and subsequent utterances, processing those messages is more efficient 
than processing verbal messages. At the same time, the continued presence of that 
text inclines one to constantly re-interpret its meaning as new text appears. This 
process understanding computer mediated text may be seen as analogous to what Ong 
(1982) would probably describe as "secondary literacy." 

On the other hand, the method has tremendous potential to increase our 
understanding of how new medi? forever change the way communication is 
understood. The increasing public demand for and access to computer-mediated 
networking systems (e.g., Internet, Prodigy), the development of electronic 
relationships, the formation of international network communities, and the building 
of information superhighways all suggest that major effort should be directed to 
cognition and mediated interaction. Through continued observation of protocol 
during mediated interaction, researchers may be able to log the transformation of a 
communicating society and the changing of a collective consciousness (Innis, 1951). 
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To sum, analysis of protocol concurrent to conversation through a computer- 
mediated channel offers a unique opportunity to map cognition during interaction. 
While there are limits to how far collected data can be generalized to face-to-face 
interaction and "ordinary" conversation, data are nonetheless quite revealing. To be 
sure, perhaps the limits of this method may not be in the shortcomings of its 
"conversation-like" features, but in holding face-to-face interaction as the ideal 
model upon which other forms of interaction are measured. 
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